Deinterlacer using both low angle and high angle spatial interpolation

ABSTRACT

A display processor integrated circuit includes a display processor portion and an on-chip programmable logic portion. The programmable logic portion can be configured to implement custom video and/or image enhancement functions. The display processor portion performs block-based motion detection. If no motion is detected for a given block of pixels, then interline gaps in the block are filled using temporal interpolation. If motion is detected, then interline gaps are filled using spatial interpolation. To maintain accuracy without unduly increasing computational complexity, a less complex high angle spatial interpolation method is employed where a low angle tilt condition is not detected. A more computationally intensive low angle spatial interpolation method can therefore be employed in low angle tilt conditions. Integrated circuit cost is reduced by employing pipelining to write parts of segment buffers at the same time that other parts are being read to perform the interpolation process.

CROSS REFERENCE TO RELATED APPLICATION

This application claims the benefit under 35 U.S.C. §120 of, and is acontinuation-in-part of, of U.S. patent application Ser. No. 10/235,628,by Chan et al., entitled “Display Processor Integrated Circuit WithOn-Chip Programmable Logic For Implementing Custom EnhancementFunctions,” filed Sep. 4, 2002 (the subject matter of theabove-identified patent application is incorporated herein byreference).

TECHNICAL FIELD

This application relates to deinterlacing of video information.

BACKGROUND INFORMATION

Video information in some video formats is interlaced. In one format, animage is displayed by painting every other line of pixels on a display,and then coming back and painting the intervening lines of pixels. Forexample, the odd scan lines are painted on the screen one by one, andthen the even scan lines are painted. The entire image is called aframe. The first set of lines is a first field, and the second set oflines is a second field. In an NTSC video signal, for example, a frameincludes 480 lines of pixels, where each line includes 720 pixels. Eachfield contains 240 lines of pixels, where each line is 720 pixels.

It is sometimes desired to increase the amount of video information byincreasing the number of pixels in each field from 240 lines to acomplete 480 lines. This process, called deinterlacing, doubles theamount of pixel information.

One technique for deinterlacing is pixel-based motion detection. See,for example, U.S. Pat. Nos. 6,166,773 and 5,473,383. Typically,pixel-based motion detection involves estimating the existence of motionfor every single pixel of the field. These estimates are based on thevalue of the pixel. In real video, an object is typically represented bya large number of pixels. Movement of objects usually appears as changesin a large group of pixels between video fields. As a result, the changeof a single pixel between fields often correlates to changes of itsneighboring pixels. This correlation is, however, generally notexploited in pixel-based motion detection. Secondly, pixel-based motiondetection generally uses information from only a few intensity valuesaround the pixel for which motion is being estimated. Most noises are,however, burst noises and pixel-based motion detection fails in burstnoise situations. Moreover, in real video a few pixels may happen to bethe same between video fields when objects with repetitive texturesmove. Pixel-based motion detection may fail in such situations becausepixel values do not change between successive video fields.

Another technique is adaptive diagonal interpolation, sometimes calleddirectional adaptive interpolation. See, for example, U.S. Pat. No.6,133,957. Directional adaptive interpolation calculates differences ofpixel pairs, selects the pair with the smallest difference, andinterpolates these pixels. Unfortunately, some real video includes pixelpatterns where there is more than one pixel pair with the smallestdifference. Moreover, the pair with the smallest difference may notgenerate the best interpolation result.

U.S. Pat. No. 5,410,356 describes another technique where motioncompensation is used for deinterlacing. In this technique, an imagefield is divided into blocks. For every block of interest, a motionestimation engine finds a group of pixels that best matches the block ina defined searching range. A displacement vector or motion vector istypically predicted that describes the spatial translation from theblock to the matched group of pixels. New lines of pixels areinterpolated using the block of interest and the matched group ofpixels. Such motion compensation techniques often do not do a job ofpredicting motion for use in deinterlacers. In real video, objects canrotate, turn and deform into other shapes. Objects can occlude eachother partially or completely. Using simple spatial displacement todetect this type of motion often results in poor decisions.Additionally, using a displacement vector to interpolate often does notobtain adequately accurate results for deinterlacing applications.Object movement cannot always be represented as an integer displacementvector. An object can move a distance to a fraction of a pixel, and thegroup of pixels corresponding to a block can be different from theblock. Finding an accurate match in such a scenario is often difficultwithout a more noise-tolerant motion detection scheme. In addition tothese problems, motion compensation is computationally complex. The costof realizing a deinterlacer using this technique may be undesirablyhigh. Pixel information is moved from a field memory to theinterpolation circuitry using line buffers. The required memorybandwidth of the pixel storing memory is also generally high, whichfurther increases system cost because higher performance memories needto be employed.

An improved deinterlacing method is sought that can be efficientlyrealized in hardware.

SUMMARY

A display processor integrated circuit (for example, for a television orfor a digital camera) includes a display processor portion and anon-chip programmable logic portion. The on-chip programmable logicportion can be configured or programmed to implement custom video and/orimage enhancement functions. The display processor portion performsblock-based motion detection. Rather than attempting to match a block ofpixels in one field with a corresponding block of pixels in a subsequentfield as is often done in motion compensation, the block-based motiondetection performed by the display processor integrated circuitgenerates a sum value and a difference value from pixel pairs incorresponding pixel locations in the block in the field preceding thefield of interest and the field subsequent to the field of interest. Ifthese sum and difference values have a predetermined relationship to oneanother, then the block is, in one particular embodiment, determined toexhibit the motion characteristic.

If the motion characteristic is not detected for a given block ofpixels, then interline gaps in the block are filled using temporalinterpolation. If, on the other hand, the motion characteristic isdetected then the interline gaps are filled using spatial interpolation.To maintain interpolation accuracy without unduly increasing complexityof the integrated circuit, a less computationally intensive high anglespatial interpolation method is employed where a low angle tiltcondition is not detected. A more accurate but more computationallyintensive low angle spatial interpolation method can therefore beadvantageously employed to interpolation in low angle tilt conditions.

The integrated circuit is designed and is specially adapted for highvolume and low production cost applications including, for example, thehigh volume consumer television market. Integrated circuit cost isreduced by reducing the cost of memories used to pass pixel data from afield memory to interpolation circuitry. In one embodiment, the memoriesinclude three segment buffers. A memory control block on the integratedcircuit retrieves new pixels to be processed from the field memory andwrites them into a part of a segment buffer at the same time that theinterpolation circuitry is reading other pixels from other parts of thesegment buffer. A certain amount of pipelining is therefore employed inthe writing and reading of the segment buffer. This pipelining increasesthe overall proportion of the time that the memory control block iswriting pixel data into the segment buffers. Because the segment buffersare receiving pixel data from the memory control block during a largerproportion of the time, the type of memory employed to realize thesegment buffer can have relaxed memory access bandwidth requirements.This allows the overall cost of the integrated circuit to be reduced.Rather than reading entire 720 pixel scan lines out of the field memoryto support the block-based motion detection and interpolation processes,smaller segments of scan lines are read out of the field memory. Thisreduces memory access bandwidth requirements on the field memory andtherefore further reduces system cost. Moreover, reading segments oflines out of the field memory rather than entire scan lines reduces theamount of memory required to pass information from the field memory tothe interpolation circuitry. This reduces system cost still further.

Multiple other advantageous aspects and embodiments are set forth in thedetailed description below. This summary does not purport to define theinvention. The invention is defined by the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a simplified diagram of the electronics of a video displaydevice in accordance with one embodiment of the present invention.

FIG. 2 is a simplified block diagram of the integrated circuit of thevideo display device of FIG. 1.

FIG. 3 is a simplified block diagram illustrating three consecutivefields of video.

FIG. 4 is a simplified diagram illustrating the overlap of the segmentsof a field.

FIG. 5 is a simplified diagram illustrating a sequence in which thesegments of a field are processed.

FIG. 6 is a simplified diagram illustrating the overlap of blocks of afield.

FIGS. 7A–7C illustrate where blocks are disposed in a segment in theleftmost column of segments, in the middle column of segments, and inthe rightmost column of segments, respectively.

FIG. 8 is a simplified flowchart that sets forth a method in accordancewith an embodiment of the present invention.

FIG. 9 is a diagram that sets forth details of the block-based motiondetection step of the method of FIG. 8.

FIGS. 10 and 11 are diagrams that set forth how high angle spatialinterpolation is performed.

FIG. 12 is a diagram of a segment that shows which pixels to begenerated are generated using interpolation, and which pixels to begenerated are not generated by interpolation.

DETAILED DESCRIPTION

FIG. 1 is a simplified system level diagram of the electronics of avideo display device 1 in accordance with one embodiment of the presentinvention. An incoming signal is received onto the video display device,for example, from an antenna 2, a coaxial cable 3, or another videosource 4. The signal passes through a tuner 5, an IF demodulator 6, ananalog-to-digital converter 7, and to a display processor 8 within anintegrated circuit 9. The display processor 8 performs deinterlacing andscaling. A programmable logic portion 10 of integrated circuit 9, eitherindependently or in concert with parts of the display processor 8,performs one or more enhancement functions. The resulting deinterlacedvideo signal is output from integrated circuit 9 to driver 11 and to adisplay device. The display device may, for example, be a cathode raytube (CRT) 12, a liquid crystal display (LCD) screen 13, a plasmadisplay 14 or other display device usable to view video. Frames of videoinformation are stored in an external RAM 15. A microcontroller 16 iscoupled to integrated circuit 9. Microcontroller 16 can control featuresand/or enhancement functions performed by integrated circuit 9. Thesefeatures and/or enhancement functions may, for example, includePicture-In-Picture (PIP), Picture-Out-Picture (POP), Cinema 1, Cinema 2,format conversion, film detection, panorama scaling, alpha blending andoverlay, VBI/Closed Captioning, On-Screen Display (OSD), and brightnessadjusting. Audio passes through audio circuitry 17 and to speaker 18.

FIG. 2 is a more detailed diagram of integrated circuit 9 of FIG. 1.Integrated circuit 9 actually has two digital video input ports 19 and20. A digital video signal received onto digital video port 19 passesthrough a format detector 21, through a FIFO 22, and to a memory controlblock 23. If a digital video signal is present on digital video port 20,then this second digital video signal passes through a second formatdetector 24, through a second FIFO 25, and to memory control block 23.In the example of FIG. 1, only one of the digital video ports, digitalvideo port 19, is used. Consecutive frames of video pass through digitalvideo input port 19, through format detector 21, through FIFO 22,through memory control block 23 and are stored in DDR SDRAM (double datarate synchronous dynamic random access memory) 15.

In one example, each frame of video is an NTSC video frame that includes480 scan lines of pixels, where each row contains 720 pixels. (A line ofpixels is sometimes called a row of pixels). In this example, a pixelinvolves two numbers: 1) an eight-bit luminance value, and 2) aneight-bit chrominance value. The chrominance value represents either red(chrominance red) color information or blue (chrominance blue) colorinformation. Whether the chrominance value is for chrominance red orchrominance blue is determined by which pixel in the line it is. Thechrominance values alternate, red and blue down the line. Whether thechrominance value is red or blue is determined by location of the pixeland is not encoded in the stored chrominance value.

Each frame is made up of two fields. The first field includes the oddscan lines of the frame. The second field includes the even scan linesof the frame. The first field includes pixels of the video image at atime before the remainder of the image represented by the pixels of thesecond field. In the example of the hardware of FIG. 2, consecutivefields of consecutive frames are received and stored field by field intoRAM 15.

It is desired to supply some video display devices with “deinterlaced”video in that the number of pixels in each field of pixels is to bedoubled. For each 240 line by 720 pixels field supplied to integratedcircuit 9, the integrated circuit is to output a frame of 480 lines by720 pixels. To convert a 240 line by 720 pixel field (called the “fieldof interest”) into a 480 line by 720 frame, corresponding blocks ofthree consecutive fields are taken out of RAM 15.

FIG. 3 illustrates three such corresponding blocks in three consecutivefields. In this example, the first field contains odd lines for a firstframe (lines 1, 3, 5 and so forth) and the second field contains evenlines (lines 2, 4, 6 and so forth) for the first frame. The third fieldis the first field containing odd lines (lines 1, 3, 5 and so forth) ofthe next frame. The middle field is the field of interest. The secondblock for which extra pixels are to be generated is a block from thisfield of interest. The number of lines of pixels within this block isdoubled. The first block is a block for the same spatial location in theframe as the second block, only the first block is from the fieldimmediately prior to the field of interest. The third block is a blockfor the same spatial location in the frame as the second block, only thethird block is from the field immediately following the field ofinterest. Motion detection block 38 of process block 26 (see FIG. 2)uses the first and the third blocks to make a determination whetherthere is motion in the second block. If, for example, an object in thevideo that happens to be in the area of the picture defined by a blockwere to move position from one field to the next to the next, thenmotion might be detected.

If motion is not detected for the area of the picture defined by thesecond block, then temporal interpolation is used to create pixels inbetween the row of pixels of the second block. In the example of FIG. 3where the second block contains even lines, the temporal interpolationprocess generates new pixels in the odd rows such that the number ofpixels in the second block is doubled. These new pixels are to fill inthe interline gaps between the even lines. These new pixels aredetermined by looking at the corresponding pixels in the first block andthird block. This is called “temporal” interpolation because pixelinformation outside the time of the field of interest (the second blockis from the field of interest) is used to interpolate and determine thenew pixels.

If, on the other hand, motion is detected within the second block, thenspatial interpolation is used to interpolate and fill in the odd linesin the second block. Spatial interpolation uses pixels in the same fieldas the second block to determine the new pixels. In this way, blockafter block within the field of interest are filled in such a way thatthe number of lines of pixels in the field of interest is increased from240 even lines to 480 odd and even lines. The interpolation andgeneration of new pixels is performed by deinterlace block 39 of processblock 26 (see FIG. 2).

The method is explained in further detail in connection with FIGS. 4–13.The blocks of pixels used in the process described above are taken outof RAM 15 in multi-block “segments” of pixels. A segment of pixels issix lines high by 288 pixels wide, except for the segments in the toprow of segments and the segments in the bottom row of segments. Segmentsin the top and bottom rows of segments are five lines high by 288 pixelswide.

FIG. 4 illustrates four segments designated A, B, C and D. Asillustrated, segment A is in the top row of segments. Segment A is fivelines high by 288 pixels wide. Segment B, which is in the second row ofsegments, is six lines high by 288 pixels wide. Segment C, which is inthe third row of segments, is six lines high by 288 pixels wide. SegmentD, which is in the bottom row of segments, is five lines high by 288pixels wide. There are three columns of segments. As illustrated in FIG.4, the right portion of the leftmost column of segments overlaps theleft portion of the middle column of segments by 64 pixels. Similarly,the right portion of the middle column of segments overlaps the leftportion of the rightmost column of segments by 64 pixels. The lastsixteen pixel positions of the segments in the rightmost column are notactually filled with pixel information because each scan line of pixelsonly contains 720 pixels.

FIG. 5 illustrates how the various rows of segments overlap one anotherin the vertical dimension. As illustrated, there are sixty rows ofsegments. Each segment overlaps the segment below it by two lines ofpixels. There are one hundred eighty total segments in a field. In FIG.5, a segment number is preceded by a number sign #. The pixels of afield stored in RAM 15 are taken out of RAM 15 and are processed in theorder illustrated in FIG. 5. The first segment of pixels to be taken outof RAM 15 and to be processed is denoted “#1”, the second segment ofpixels to be taken out of RAM 15 and to be processed is denoted “#2”,and so forth.

Three segments of pixels, one from the field preceding the field ofinterest, one in the field of interest, and one following the field ofinterest, are taken out of RAM 15 by memory control block 23 (see FIG.2), and are passed to a process block 26 via three respective segmentbuffers 27, 28 and 29. Each of segment buffers 27 and 29 is a bufferthat stores six rows of 288 pixels each. Segment buffer 28 is smaller.It is a buffer that stores five rows of 288 pixels. Each of segmentsbuffers 27, 28 and 29 is coupled to process block 26 by its own 128-bitwide bus.

As the blocks of the segment of the field of interest in segment buffer28 are processed, the interpolated new pixels are written into a FIFO 30(see FIG. 2). FIFO 30 is coupled to processor block 26 by a 128-bit widebus. FIFO 30 is coupled to memory control block 23 by another 128-bitwide bus. Noise reduction results are output by noise reduction block 40of process block 26 (see FIG. 2) and are supplied to memory controlblock 23 via FIFO 41.

Once the interpolation process is completed for all the blocks of thesegment, then FIFO 30 contains all the newly interpolated pixels for theblocks in that segment. These blocks of newly interpolated pixels arestored by memory control block 23 in RAM 15. When the resulting field of“deinterlaced” video is to be output, then the segment of newlyinterpolated pixels is combined with the original segment and theresulting “deinterlaced” segment of blocks is output onto output bus 31to FIFO 32. Each pixel is represented by 16 bits, and 8 pixels (all thepixels, both original and interpolated, in a column of the segment) areoutput onto bus 31 at the same time. Output bus 31 is therefore 128 bitswide. FIFO 32 contains 960 such 128-bit wide words.

The deinterlaced lines of video pass through FIFO 32, through scalarblock 33, through PGA (Programmable Gate Array) block 10, and are outputfrom integrated circuit 9. Each pixel is sixteen bits as it is outputfrom scalar block 33. Pixels are output by scalar block 33 and aresupplied to PGA block 10, pixel by pixel on a 16-bit bus. Numerousdifferent PLD and FPGA architectures can be employed to realize PGAblock 10. The use of the term FPGA architecture here describes theoverall logic block and interconnect architecture and does notnecessarily imply any particular configuration bit storage mechanism.PGA block 10 is to be factory-customized by the integrated circuitmanufacturer or television manufacturer, and is not to be programmed inthe “field” by an end-user of a television. In one example, PGA block 10is customized so that it implements a customer-specific videoenhancement function by changing just one mask. For details on oneparticular example of PGA block 10, see U.S. patent application Ser. No.10/235,628, by Chan et al., entitled “Display Processor IntegratedCircuit With On-Chip Programmable Logic For Implementing CustomEnhancement Functions,” filed Sep. 4, 2002 (the subject matter of whichis incorporated herein by reference).

The deinterlaced video may, or may not, take pass through an enhancementblock 34. Whether the deinterlaced video passes through enhancementblock 34 is determined by PGA 10. An example of an enhancement performedby enhancement block 34 is brightness adjustment. Each pixel is 24 bitswide, and pixels come out of enhancement block 34 pixel by pixel on a24-bit wide bus to PGA block 10. If an analog video output signal isdesired, then the deinterlaced video stream passes through adigital-to-analog converter (DAC) block 35 and is output from integratedcircuit 9 in analog form.

FIG. 6 illustrates how the blocks of pixels overlap one another in thevertical and horizontal dimensions. The top left block of pixels,designated in FIG. 6 as block A, is five pixel lines high as are all theother blocks of the top row of blocks. This block A is eighteen pixelswide, as are all the other blocks of the leftmost column of blocks.

The bottom right block of pixels, designated in FIG. 6 as block Z, isfive pixel lines high as are all the other blocks of the bottom row ofblocks. This block Z is eighteen pixels wide, as are all the otherblocks of the rightmost column of blocks. All the other blocks of thefield that are not in the top or bottom rows and are not in the leftmostor rightmost columns are six pixels high by twenty pixels wide. Asillustrated, a block of pixels overlaps a block of pixels above it bytwo lines of pixels. As illustrated, a block of pixels overlaps a blockof pixels to its right by four lines of pixels. Accordingly, it is seenthat a 288-pixel wide segment of the leftmost column of segments (seeFIG. 4) can store more pixels of information than are represented by theleftmost sixteen blocks of pixels.

FIG. 7A illustrates how the pixels of the leftmost sixteen blocks ofpixels are stored in a segment in the leftmost column of segments. Ascan be seen from FIG. 7A, although 288 pixels wide of information isread out of RAM 15, the rightmost thirty pixels are pixels to the rightof the end of the sixteenth column of blocks. Any blocks of pixels inthis last thirty columns of pixels are not processed with the otherblocks of the segment, but rather are processed with the blocks of thenext segment to the right.

FIG. 7B illustrates how the pixels of block-columns seventeen throughthirty are stored in the segment to the right of the segment of FIG. 7A.As illustrated, the leftmost thirty columns of pixels are to the left ofblock#17. Blocks of pixels in these leftmost thirty pixel columns havetherefore been processed previously with the segment of FIG. 7A. Theblocks of pixels to the left of block#17 are therefore not processedwith block#17 through block#30 of the current segment. Similarly, blocksto the right of block#30 are not to be processed with the blocks ofblock-columns 17–30, but rather are to be processed with the segment tothe right. There are thirty columns of pixels to the right of block#30.

FIG. 7C illustrates how the blocks of the segment to the right arestored. As illustrated, blocks in the leftmost thirty columns of pixelsare not processed because they were processed previously as blocks ofthe segment of FIG. 7B. Fifteen blocks (block#31 through block #45) areprocessed. The rightmost sixteen columns of pixels are located to theright of block#45. These columns of pixels do not store pixelinformation because these pixels are beyond the right edge of the frame.

FIG. 8 illustrates a method 100 in accordance with the presentinvention. As set forth above in connection with FIG. 5, segments areprocessed one by one in the order indicated in FIG. 5. Accordingly, thefirst segment (segment#1) of the field of interest is loaded (step 101)into segment buffer 28. Moving block to block from left to right acrossthe blocks of the segment, the decision (step 102) is made whethermotion is detected for each block. One particular motion test is setforth in FIG. 9. In this motion test, the luminance values of eachsuccessive pair of corresponding pixels in the field immediatelypreceding the field of interest and corresponding pixel in the fieldimmediately subsequent to the field of interest are averaged, and theseaverages are summed together for all the pixels of the block. This valueis called “SUM” in FIG. 9. Similarly, the difference is taken of theluminance values of each successive pair of corresponding pixels in thefield immediately preceding the field of interest and correspondingpixel in the field immediately subsequent to the field of interest, andthese differences are summed together for all the pixels of the block.This value is called “DIFF” in FIG. 9. If DIFF is greater than theproduct of SUM and a threshold ratio, then the decision is made that theblock exhibits motion.

The results of these motion detection tests are stored in a motionhistory buffer. Each bit in the motion history buffer represents whethermotion has been detected for a corresponding block. In one example, themotion history buffer includes one bit for each block of the currentsegment. In another example, the motion history buffer is bigger andincludes a bit for each block in the current field such that the motionhistory buffer contains an array of 45 by 60 bits, one bit for each of45 by 60 blocks in a field.

After the results of the motion detection tests for an entire segment ofblocks are stored in the motion history buffer, the leftmost block ofthe segment is considered (step 103). If no motion was detected for thisblock (step 104), then the upper row of missing pixels to be generatedfor the block are determined (step 105) using temporal interpolation.If, for example, a pixel in line three of the field of interest is to bedetermined (see FIG. 3), then the luminance value of the correspondingpixel from line three in the preceding field is averaged with theluminance value of the corresponding pixel from line three in thesubsequent field. The resulting average is the generated pixel value.This pixel value is placed into line three in the segment in FIFO 30(see FIG. 2). This process is repeated for each pixel to be generated inthe row of pixels to be generated in the current block. The newlygenerated pixels are stored in FIFO 30.

If, on the other hand, motion was detected for the current block, thenthe upper row of pixels to be generated for the block is determined(step 106) using spatial interpolation. One of two types of spatialinterpolation could be used to generate a particular pixel of interest,either high angle spatial interpolation or low angle spatialinterpolation. A determination is made whether low angle spatialinterpolation will be used (as explained further below in connectionwith low angle spatial interpolation). If this determination indicatesthat low angle spatial interpolation will not be used, then the pixel ofinterest is determined using high angle spatial interpolation. In thisexample, performing temporal interpolation in step 105 is lesscomputationally intensive than performing spatial interpolation in step106.

FIGS. 10 and 11 illustrate how high angle spatial interpolation isperformed. In FIG. 10, the “X” represents the missing pixel to begenerated. The pixel is in the (i-1)th line. In the example of thesecond block in the field of interest in FIG. 3, this pixel might be inline 3, for example. Pixels in the line above pixel X are designatedpixels P, A, B, C and Q as illustrated. Pixels in the line below pixel Xare designated pixels R, D, E, F and S as illustrated. Each pixel valueincludes an eight-bit chrominance value and an eight-bit luminancevalue. The luminance and chrominance values of pixel X are determined inaccordance with the steps set forth in FIG. 11. The values A through Fin the VERT_GRAD and HORI_GRAD equations are luminance values. For theother equations of FIGS. 10 and 11, pixels values used in luminance,X_(L) equations are luminance values, whereas pixel values used inchrominance Xc equations are chrominance values.

The process of determining a pixel using low angle spatial interpolationinvolves three overall steps. First, a set of gradient values isdetermined. Second, the gradient values are examined to determinewhether there exists in the gradients a pattern indicative of left tiltor a pattern indicative of right tilt. Third, either left tiltinterpolation or right tilt interpolation is performed depending onwhether the tilt detected is left tilt or right tilt.

Gradient values are determined by considering a group of pixels in ablock. The group contains four rows of pixels, and where each rowcontains twenty-three pixels. The naming convention set forth inconnection with FIG. 10 is employed to consider each set of six pixels.From one such set of six pixels, a gradient value X is determined for aninterline gap pixel of interest. This gradient value X is determined inaccordance with the equation A–D+2*(B–E)+C–F using luminance values ofeach of these pixels. If the result of this gradient calculation isgreater than a predetermined threshold value, then the value X isdetermined to be a digital “1”. If, on the other hand, the result ofthis calculation is less than the predetermined threshold value, thenthe value X is determined to be a digital “0”. For each gradient value,a sign value is also stored where the sign bit indicates whether thegradient value is negative or positive. In this way, the template of sixpixels A–F (see FIG. 10) is moved top to bottom down the leftmost threecolumns of the four rows of pixels in the block (the block is in thefield of interest). This generates three gradient values for theleftmost three columns of pixels. This process repeats column by columnto the right through the block, such that for every column, threegradient values are determined. Each gradient is stored with anassociated sign value.

The successive sets of three gradients are examined to look for a firstpattern. The first pattern involves at least four gradients in the toprow of gradients being digital ones, but where the correspondinggradients below those in the next row down are all digital zeros, andthe corresponding gradients below those in the next row down are alldigital zeros. If such a first pattern of gradients is found, then moregradients are determined and examined to see if a second pattern existsto the right of the first pattern. The second pattern exists wherenumber G of consecutive gradients in the top row of gradients aredigital ones, where the G gradients below those in the next row down oneall digital ones, and where the G gradients below those in the next rowdown are all digital zeros. Number G can be set to be, for example,minimum of two. If this second pattern is found, then more gradients aredetermined and examined to see if a third pattern exists to the right ofthe second pattern. This third pattern exists where number H consecutivegradients in the top row of gradients are all digital zeros, where the Hgradients below those in the next row down are all digital ones, andwhere the H gradients below those in the next row down are all digitalzeros. Number H can be set to be, for example, in the range of four toseventeen. If this pattern is found, then the number of consecutive onesin the second row of this pattern is stored. If this third pattern ofgradients is found, then more gradients are determined and examined tosee if a fourth pattern exists to the right of the third pattern. Thefourth pattern exists where at least four consecutive gradients in thetop row of gradients are digital zeros, where the correspondinggradients below those in the next row down one all digital zeros, butwhere the corresponding gradients below those in the next row down areall digital ones. If these four patterns are found in order from left toright in the set of gradients, and if all the digital ones have the samesign, then a determination is made that a “left tilt” exists. Left tiltluminance low angle spatial interpolation is performed by taking thenumber stored when the third pattern was detected, and dividing thisnumber by two. If, for example, the number stored when the third patternwas detected was seven, then the result of dividing by two yields thevalue of three (plus a remainder which is discarded). The pixel in therow above the pixel of interest but three pixels to the left is averagedwith the pixel in the row below the pixel of interest but three pixelsto the right. This average is the left tilt luminance low angle spatialinterpolation result.

If left tilt luminance low angle spatial interpolation is not performed,then the same process is repeated to look for the conditions of righttilt luminance low angle spatial interpolation. The first pattern to belooked for at the left of the set of gradients exists where at leastfour consecutive gradients in the top row of gradients are digitalzeros, where the corresponding gradients below those in the next rowdown are all digital zeros, but where the corresponding gradients belowthose in the next row down are all digital ones. The second pattern tobe looked for to the right of the first pattern exists where the numberG consecutive gradients in the top row of gradients are digital zeros,where G gradients below those in the next row down are all digital ones,and where G gradients below those in the next row down are all digitalzeros. Number G can be from 4 to 17. If this pattern is found, thenumber of consecutive ones in the second row of this pattern is stored.The third pattern to be looked for to the right of the second patternexists where H consecutive gradients in the top row of gradients aredigital ones, where H gradients below those in the next row down are alldigital ones, and where H gradients below those in the next row down areall digital zeros. Number H can be set to, for example, minimum of two.The fourth pattern to be looked for to the right of the third patternexists where at least four consecutive gradients in the top row ofgradients are digital ones, where the corresponding gradients belowthose in the next row down are all digital zeros, and where thecorresponding gradients below those in the next row down are all digitalzeros. If these four patterns are detected, and if all gradient digitalone values have the same sign, then right tilt luminance low anglespatial interpolation is performed. The number stored when the secondpattern was detected is divided by two. If, for example, the numberstored when the third pattern was detected was seven, then the result ofdividing by two yields the value of three (plus a remainder which isdiscarded). The pixel in the row above the pixel of interest but threepixels to the right is averaged with the pixel in the row below thepixel of interest but three pixels to the left. This average is theright tilt luminance low angle spatial interpolation result.

Chrominance low angle spatial interpolation uses the same calculationsas the chrominance high angle spatial interpolation set forth in FIG.11. Left tilt chrominance low angle spatial interpolation is determinedin accordance with Xc=(P+S)/2. Right tilt chrominance low angle spatialinterpolation is determined in accordance with Xc=(Q+R)/2. The relativelocations of pixels P, Q, R, S with respect to pixel X (the pixel ofinterest) is indicated in FIG. 10. The P, Q, R and S pixels values usedare chrominance pixel values.

If the examination of gradient values results in neither left tilt norright tilt low angle spatial interpolation being performed, then highangle spatial interpolation is performed. FIG. 11 illustrates how highangle spatial interpolation is performed.

Returning to the flowchart of FIG. 8, after the upper row of missinginterline gap pixels has been generated for the current block (usingeither the temporal interpolation, or low angle spatial interpolation,or high angle spatial interpolation), then processing proceeds todecision block 107. If the block being processed is not the last blockof the current segment (the rightmost block), then the next block to theright is considered (step 108). The upper row of missing interline gappixels is determined (steps 104–106) for this next block, and theprocess continues until the last block (on the right edge) of thesegment buffer is processed. In this way, the upper row of missinginterline gap pixels is determined for each block of the currentsegment.

Once the upper row of missing interline gap pixels has been determinedfor the last block of the segment, a decision is made (step 109) whetherthe just generated row of interline gap pixels was the last suchinterline gap row to be generated for the segment. If it was not thelast interline gap row, then the next row down of missing interline gappixels is considered (step 110). The previous steps (steps 104–108) areused to fill in this next row of interline gap pixels for all the blocksof the segment. This process continues for each successive one of themissing rows of interline gap pixels.

When the last missing row (the bottom missing interline gap pixel row)of the last block of the segment (the rightmost block of the segment)has been created, then a decision (step 111) is made whether the segmentjust processed was the last segment of the field. If it was not, thenthe next segment is considered (step 112) and processing returns to step102. In this manner, missing interline gap pixels are generated for eachsuccessive segment of the field of interest. The order of processing ofsegments is as set forth in FIG. 5.

Once the last segment has been processed, then processing goes to (step113 of FIG. 8) the next field and the process repeats. Accordingly, thenumber of pixels in the field is increased from 240 scan lines of 720pixels each, to 480 scan lines of 720 pixels each. The newly generatedpixels are placed into FIFO 30.

Pipelined Segment Buffer Loading:

As set forth above in connection with FIG. 8, a decision (step 102) ismade for each block of a segment whether motion has been detected. Then,after this decision has been made for all the blocks of the segment, thefirst missing row of pixels is generated for each of the blocks of thesegment. Then the next missing row of pixels is generated for each ofthe blocks of the segment. In this manner, rows of missing pixels aregenerated, row by row, from top to bottom. The high angle interpolationuses one pixel above the pixel to be generated and one pixel below thepixel to be generated. The low angle interpolation uses pixels in tworows above the pixel to be generated and pixels in two rows below thepixel to be generated. Accordingly, once the second row of missinginterline gap pixels has been generated for the blocks of the currentsegment, the top row of original pixels of the segment is no longerrequired for the generation of new pixels. This is true regardless ofwhether low angle spatial interpolation is used or not.

Accordingly, once the second row of missing interline gap pixels hasbeen generated for the blocks of the current segment, memory controlblock 23 overwrites the upper line of original pixels in segment buffers27 and 29 with a new line of pixels with the upper line of originalpixels in the next segment to be processed. Similarly, the first row ofpixels in segment buffer 28 can be overwritten with a new line of pixelsin the first row of the next segment to be processed.

Then again, once the third row of missing interline gap pixels has beengenerated for the blocks in the current segment, then memory controlblock 23 overwrites the second line of original pixels in the currentsegment buffer 27 and 29 with the second line of pixels for the nextsegment to be processed. Similarly, the second row of pixels in segmentbuffer 28 is overwritten with a new line of pixels in the second row ofpixels of the next segment to be processed.

This manner of pipelined loading of segment buffers 27, 28 and 29 (theoverwriting of just used but no longer needed pixel data with new linesof pixel data at the same time that other pixel data is being read outof the segment buffers and is being used to perform interpolation)reduces memory bandwidth requirements of the external memory bus andallows a lower operating frequency of the memory control block 23 andthe external memory block 15. This eases design reguirements on memorycontrol block 23 and reduces system cost by allowing the use of lowcost, low performance external DRAMs. Segment buffers 27–29 aredual-port memories so that memory control block 23 can write originalpixel data into the segment buffers at the same time that process block26 reads pixel data out of the segment buffers.

Boundary Conditions:

Where the block of interest is in the upper line of blocks of an evenfield, the above method is not performed. Rather, a pixel in this topline of such a block is generated by averaging a corresponding pixel inthe immediately preceding field and in the immediately following field.This is done for each pixel in line number one at the top of the field,regardless of whether motion is detected or not.

Similarly, where the block of interest is in the bottom row of blocks ofan odd field, the above method is not performed. Rather, a pixel in thebottom line of such a block is generated by averaging a correspondingpixel in the immediately preceding field and in the immediatelyfollowing field. This is done for each pixel in line number 480 at thebottom of the field, regardless of whether motion is detected or not.

Where the block of interest is at the left edge of the field and thepixel to be generated is at the left edge of the block, then temporalinterpolation is always used, regardless of whether motion is detectedor not. Similarly, if the block of interest is at the right edge of thefield and the pixel to be generated is at the right edge of the block,then temporal interpolation is always used, regardless of whether motionis detected or not.

As set forth above, low angle spatial interpolation requires pixels intwo lines above the pixel to be generated as well as pixels in the twolines below the pixel to be generated. Accordingly, low angle spatialinterpolation cannot be performed to generate pixels in the upper row ofpixels to be generated in a block. Similarly, low angle spatialinterpolation cannot be performed to generate pixels in the bottom rowof pixels to be generated in a block. Low angle spatial interpolationalso requires pixels in two columns of pixels to the left of the pixelto be generated as well as pixels in two columns of pixels to the rightof the pixel to be generated. Accordingly low angle spatialinterpolation cannot be performed to generate pixels in the leftmost twocolumns of pixels of a block, nor in the rightmost two columns of pixelsof a block.

As set forth above, high angle spatial interpolation requires pixels inthe column to the left of the pixel to be generated. Accordingly, highangle spatial interpolation is therefore not performed to generatepixels in the leftmost column of pixels of the block. Similarly, highangle spatial interpolation requires pixels in the column to the rightof the pixel to be generated. Accordingly, high angle spatialinterpolation is not performed to generate pixels in the rightmostcolumn of pixels of the block.

In the boundary situations above, if low angle spatial interpolation isotherwise called for but cannot be performed due to a boundarycondition, then high angle spatial interpolation is used. If one of lowor high angle spatial interpolation is called for but neither type ofspatial interpolation can be performed due to a boundary condition, thentemporal interpolation is used.

FIG. 12 illustrates the pixels of a block. The block is twenty pixelswide and includes six rows of original pixels. The six rows of originalpixels are illustrated, as are the five rows of interline gap pixels tobe generated. Which pixels to be generated cannot be generated by highangle and low angle spatial interpolation are designated.

Although the present invention is described in connection with certainspecific embodiments for instructional purposes, the present inventionis not limited thereto. Although the motion detection block 38, thedeinterlacer block 39 and the noise reduction block 40 are illustratedas separate blocks of circuitry in FIG. 2, the functionality of theseblocks need not be so separated. The motion detection and deinterlacingblocks of the process block 26 of FIG. 2 can, in one embodiment, beembodied by describing the functionality of the motion detection anddeinterlacing blocks in verilog or another hardware descriptionlanguage, and then using commercially available hardware synthesissoftware to generate hardware that realizes the function in integratedcircuit form. In such an embodiment, the motion detection anddeinterlacing circuitry would likely be highly intermixed and would notbe readily recognizable as separate blocks of circuitry. Memory accessbandwidth requirements on the memories used to pass pixel data from thefield memory to the interpolation circuitry can be reduced in ways otherthan writing new pixel data into one part of a segment buffer at thesame time that other pixel information in the segment buffer is beingaccessed by the interpolation circuitry. A pair of ping-pong segmentbuffers may, for example, be used such that the memory control blockwrites new pixel data into a first segment buffer at the same time thatthe interpolation circuitry uses pixel data in a second segment buffer.When the interpolation circuitry has examined all the pixel data in thesecond segment buffer, then memory control block starts writing newpixel data into the second segment buffer and the interpolationcircuitry starts examining pixel data in the first segment buffer. Inthis way, the uses of the two segment buffers switch in a ping-pongmanner. Other known techniques for decreasing memory access bandwidthrequirements can be used. Although the spatial interpolation methods setforth above interpolate using pixel data from only one field (the fieldof interest), this need not be the case. A spatial interpolation methodmay also use a relatively small amount of pixel data from other fieldsand still be a spatial interpolation method. Similarly, a temporalinterpolation method may use a relatively small amount of pixel datafrom pixels in the field of interest around the missing pixel beinginterpolated and still be a temporal interpolation method. Theblock-based motion detection determination may, in some embodiment,determine the relative amounts of spatial and temporal interpolationperformed.

In some embodiments, PGA block 10 interfaces to the memory control block23 such that PGA block 10 accesses portions of the fields of pixels inRAM 15 that are not being used by the other blocks of FIG. 2. Memorycontrol block 23 may, for example, involve a microcoded DMA statemachine that receives and executes DMA commands from a microcode controlstore. In ordinary operation, the DMA state machine executes DMAcommands such that the DMA state machine carries out the loading ofsegment buffers 27–29 as set forth in the description of interpolationabove. PGA block 10 is, however, also able to write DMA commands into toa part of the microcode control store, thereby allowing PGA block 10 tocause a DMA transfer of an amount of pixel data from RAM 15 to PGA block10. PGA block 10 can then manipulate the pixel data. PGA block 10 cancause the resulting pixel data to be written back into RAM 15 by writingan appropriate DMA command into the control store. The DMA state machineexecutes the DMA command, thereby retrieving the pixel data from PGAblock 10 and places it back into RAM 15 at a location identified by theDMA command. Although a DMA technique is set forth here by which PGAblock 10 can access pixel data in RAM 15, other techniques for givingPGA block 10 access to this pixel data can be employed. Accordingly,various modifications, adaptations, and combinations of various featuresof the described embodiments can be practiced without departing from thescope of the invention as set forth in the following claims.

1. A method of generating missing pixels to fill in interline gaps in afield of pixels of video information, the method comprising: (a)determining whether pixels surrounding a missing pixel exhibit apredetermined gradient characteristic, and if the surrounding pixelsexhibit the predetermined gradient characteristic then storing a firstmissing pixel gradient value associated with the missing pixel, whereasif the pixels surrounding the missing pixel do not exhibit thepredetermined gradient characteristic then storing a second missingpixel gradient value associated with the missing pixel; (b) repeatingstep (a) for each of a plurality of missing pixels of a block of missingpixels and thereby generating a plurality of missing pixel gradientvalues; (c) examining the plurality of missing pixel gradient valuesgenerated in (a) and (b) and determining whether a first predeterminedpattern of missing pixel gradient values exists in the plurality ofmissing pixels, the first predetermined pattern of missing pixelgradient values being indicative of a first tilt gradient condition; (d)if the first predetermined pattern of missing pixel gradient values isdetermined to exist in (c) then applying a first tilt spatialinterpolation method to generate each of the missing pixels of the blockof missing pixels; (e) if the first predetermined pattern of missingpixel gradient value is not determined to exist in (c) then examiningthe plurality of missing pixel gradient values generated in (a) and (b)and determining whether a second predetermined pattern of missing pixelgradient values exists in the plurality of missing pixels, the secondpredetermined pattern of missing pixel gradient values being indicativeof a second tilt gradient condition; and (f) if the second predeterminedpattern of missing pixel gradient values is determined to exist in (e)then applying a second tilt spatial interpolation method to generateeach of the missing pixels of the block of missing pixels.
 2. The methodof claim 1, wherein the plurality of missing pixel gradient valuesexamined in (c) includes at least three missing pixel gradient valuesbut less than fifteen missing pixel gradient values, and wherein theplurality of missing pixel gradient values examined in (e) includes atleast three missing pixel gradient values but less than fifteen missingpixel gradient values.
 3. The method of claim 1, wherein the first tiltspatial interpolation method is a left tilt spatial interpolationmethod, and wherein the second tilt spatial interpolation method is aright tilt spatial interpolation method.
 4. The method of claim 1,further comprising: (g) before step (a) determining whether a block ofsaid pixels of video information exhibits a motion characteristic; and(h) if neither the first predetermined pattern is determined to exist instep (c) nor the second predetermined pattern is determined to exist instep (e), and if the block of pixels is determined in step (g) toexhibit the motion characteristic, then applying a high angle spatialinterpolation method to generate each of the missing pixels of the blockof missing pixels.
 5. The method of claim 1, wherein the first tiltspatial interpolation method uses more than two rows of pixels of thefield, and wherein the second tilt spatial interpolation method usesmore than two rows of pixels of the field.
 6. The method of claim 1,wherein the determining of (a) involves using at least three pixelluminance values from pixels in a first row of pixels of the field andalso involves using at least three pixel luminance values from pixels ina second row of pixels of the field, and wherein the missing pixel is tofill in an interline pixel gap between the first row of pixels and thesecond row of pixels.
 7. A method of generating missing pixels to fillin interline gaps in a field of pixels of video information, the methodcomprising: (a) using temporal interpolation to generate a firstplurality of the missing pixels; (b) using a high angle spatialinterpolation method to generate a second plurality of the missingpixels, wherein the high angle spatial interpolation method uses anumber A of the pixels of video information to generate a missing pixel;and (c) using a low angle spatial interpolation method to generate athird plurality of the missing pixels, wherein the low angle spatialinterpolation method uses a number B of the pixels of video informationto generate a missing pixel, wherein B is greater than A, and whereinthe low angle spatial interpolation method involves: (c1) generating ablock of gradient pixel values; and (c2) examining the block of gradientpixel values and determining whether either a left tilt gradientcondition or a right tilt gradient condition is exhibited by the blockof gradient pixel values, wherein if the left tilt gradient condition isexhibited then performing a left tilt low angle spatial interpolationmethod to generate the third plurality of missing pixels, and wherein ifthe right tilt gradient condition is exhibited then performing a righttilt low angle spatial interpolation method to generate the thirdplurality of missing pixels.
 8. The method of claim 7, furthercomprising: determining whether a block of said pixels of said fieldexhibits a motion characteristic, wherein if the block is determined notto exhibit the motion characteristic then using the temporalinterpolation of step (a) to generate missing pixels to fill interlinegaps in the block, and wherein if the block is determined to exhibit themotion characteristic then using either the high angle spatialinterpolation method of step (b) or the low angle spatial interpolationmethod of step (c) to generate missing pixels to fill the interline gapsin the block.
 9. The method of claim 8, wherein high angle spatialinterpolation is performed to generate interline gap pixels for theblock if the block exhibits the motion characteristic and if neither theleft tilt gradient condition is exhibited in step (c2) nor if the righttilt gradient condition is exhibited in step (c2).
 10. The method ofclaim 1, further comprising: (g) if neither the first predeterminedpattern is determined to exist in step (c) nor the second predeterminedpattern is determined to exist in step (e), then applying a low anglespatial interpolation method to generate each of the missing pixels ofthe block of missing pixels.
 11. The method of claim 1, wherein theblock of missing pixels has a dimension of fewer than twenty-four pixelsby fewer than seven pixels.
 12. The method of claim 7, wherein the lefttilt low angle spatial interpolation method is a left tilt luminance lowangle spatial interpolation method, and wherein the right tilt low anglespatial interpolation method is a right tilt luminance low angle spatialinterpolation method.
 13. The method of claim 7, wherein the left tiltlow angle spatial interpolation method is a left tilt chrominance lowangle spatial interpolation method, and wherein the right tilt low anglespatial interpolation method is a right tilt chrominance low anglespatial interpolation method.
 14. The method of claim 8, wherein highangle spatial interpolation is performed to generate interline gappixels for the block if the block exhibits the motion characteristic andif there are less than two lines of pixels either above or below theinterline gap pixels that are to be filled in.